System and method for accepting disparate types of user input

ABSTRACT

A user interface allows a user to input handwritten, key-press, and spoken text in a seamless, synchronized manner. A text input panel accepts soft keyboard presses and handwritten words, characters, and gestures. A text recognizer/synchronizer integrates textual input from various sources while recognizing and preserving the order in which a user entered text via the soft keyboard, via handwriting, and/or by speaking. Synchronized text may be displayed in a stage area of the text input panel before being passed to an operating system message router and/or an application program. While in handwriting recognition mode, various permutations and combinations of a word recognition area, a character recognition area, and a keybar/keypad may optionally be displayed.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of the Nov. 10, 2000 filingdate of copending provisional application, the serial number of whichhas not yet been assigned, entitled System And Method For AcceptingDisparate Types Of User Input, attorney docket number 03797.00074, whichis incorporated herein by reference.

[0002] The present application is related to (attorney docket number3797.00066), entitled Method and Apparatus For Improving the Appearanceof Digitally Represented Handwriting; to (attorney docket number3797.00067), entitled Highlevel Active Pen Matrix; to (attorney docketnumber 3797.00069), entitled Selection Handles in Editing ElectronicDocuments; to (attorney docket number 3797.00070), entitled InsertionPoint Bungee Space Tool; to (attorney docket number 3797.00072),entitled Simulating Gestures of a Mouse Using a Stylus and ProvidingFeedback Thereto; to (attorney docket number 3797.00075), entitled InAir Gestures; to (attorney docket number 3797.00076), entitled MouseInput Panel Windows Class List; to (attorney docket number 3797.00077),entitled Mouse Input Panel and User Interface; to (attorney docketnumber 3797.00079), entitled System and Method For Inserting ImplicitPage Breaks, each of which was filed Nov. 10, 2000 and each of which isincorporated herein by reference.

BACKGROUND OF THE INVENTION

[0003] As used herein the term Tablet PC refers to, by way of example,without limitation, a full-function “MICROSOFT” “WINDOWS” operatingsystem-based personal computer incorporating various convenient andintuitive aspects of pencil and paper into a user's interaction with aPC. Such Tablet PCs provide a very natural way to interact with acomputer, namely, by writing on it, without sacrificing the power orutility of various “WINDOWS” operating systems and/or various “WINDOWS”operating system-based desktop applications. As will be apparent, TabletPCs, as the term is used herein, are not limited to computers running“WINDOWS” operating system. Instead, a Tablet PC may also run any otherknown, or later-developed, operating system.

[0004] Computer users are generally familiar with interacting withconventional personal computers via a mouse and keyboard. Tablet PC's,however, generally do not have a physical keyboard or a mouse as part oftheir user interfaces. Instead, the user will often interact with aTablet PC in the form of inputting textual information, navigating amongdocuments, entering commands, and the like, by using a pen or stylus onthe Tablet PC's display.

[0005] Conventional stylus and tablet user interfaces have variousshortcomings, though. For instance, such conventional user interfacesare typically cumbersome to use, limit a user to one mode of inputtingtext information at a time, and require that the user manually switchbetween each of the various input modes.

[0006] Tablet PC's are preferably adapted for portability and use in avariety of settings, including, but not limited to, meetings withmultiple attendees and in relatively secluded settings, such as in auser's office or home. Because handwritten notes are more appropriate ina meeting setting, while speech may be more efficient for entering textinto correspondence from a relatively private setting, an integratedspectrum of various ways to enter textual information into a Tablet PCis desirable. Accordingly, there is a need for an improved computer userinterface that allows users of Tablet PC's to input multimedia textinformation (including, but not limited to, handwritten text, spokentext, and/or virtual keyboard-entered text) in various ways, whileminimizing the required amount of manual switching between input modes.

SUMMARY OF THE INVENTION

[0007] A system and method in accordance with certain inventiveprinciples overcomes the foregoing shortcomings of conventional stylusand tablet user interfaces by providing a text input panel (“TIP”) foraccepting soft keyboard presses and handwritten words, characters, andgestures. A text input panel in accordance with various principles ofthe invention allows a user to input handwritten, key-press, and spokentext in a seamless, synchronized manner that takes advantage of thebenefits of each manner of inputting text. Such benefits may depend uponthe effectiveness and/or the appropriateness of using a particular inputmethod under certain circumstances.

[0008] A text recognizer/synchronizer integrates textual input fromvarious sources while recognizing and preserving the order in which auser entered text via the soft keyboard, via handwriting, and/or byspeaking. Synchronized text may be displayed in a stage area of the textinput panel before being passed to an operating system message routerand/or an application program. While in handwriting recognition mode,various permutations and combinations of a word recognition area, acharacter recognition area, and a keybar/keypad may optionally bedisplayed.

[0009] Additional features and advantages of the invention will beapparent upon reviewing the following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010]FIG. 1 is a schematic block diagram of a conventionalgeneral-purpose digital computing environment that can be used toimplement various aspects of the present invention.

[0011]FIG. 2 illustrates a Tablet PC that can be used in accordance withvarious aspects of the present invention.

[0012]FIG. 3 is a simplified schematic block diagram depicting data flowamong various input recognizers and other components in accordance withcertain inventive principles.

[0013]FIG. 4 depicts several example layouts for a slide out text inputpanel (“TIP”) having three tabs for switching between input recognizers.

[0014]FIG. 5 depicts several example layouts for the Word recognizerarea of FIG. 4.

[0015]FIG. 6 depicts a text input panel docked at the taskbar.

[0016]FIG. 7 depicts a text input panel docked at the taskbar as in FIG.6 with a character recognizer area.

[0017]FIG. 8 depicts the text input panel docked at the taskbar as inFIG. 6 with the keyboard displayed.

[0018]FIG. 9 depicts the text input panel docked at the taskbar as inFIG. 6 with a toggle-on button in the lower right-hand corner and akeybar as a button pad to the right.

[0019]FIG. 10 depicts the character recognizer area on and as a singlepad.

[0020]FIG. 11 depicts the docked text input panel with a multi-linestage.

[0021]FIG. 12 depicts a docked text input panel with a fall widthkeyboard.

[0022]FIG. 13 depicts an example layout with a docked TIP, having thekeybar shown as a keypad on the left-hand side.

[0023]FIG. 14 depicts an example layout with a docked TIP having acharacter recognizer area, also referred to as a jot, on the left-handside.

[0024]FIG. 15 depicts an example layout with a docked TIP having amulti-line stage to the left and Keyboard and Writing tabs to the right.

[0025]FIG. 16 depicts an example layout with a docked TIP below thetaskbar.

[0026]FIG. 17 depicts an example layout with a docked TIP having adouble-line word recognizer area and a double-stacked keybar and stageto the left.

[0027]FIG. 18 depicts an example layout similar to the example layout inFIG. 17 and depicting a character recognizer area to the left of theword recognizer area.

[0028]FIG. 19 depicts an example layout similar to the example layout inFIG. 17 with the keyboard shown instead of the word recognizer area.

[0029]FIG. 20 depicts an example layout with a docked TIP having a stageat the bottom and a keybar, in the form of a keypad, to the left.

[0030]FIG. 21 depicts an example layout similar to the example layout inFIG. 20 with a character recognizer area shown.

[0031]FIG. 22 depicts an example layout with a keyboard shown and astage area at the bottom.

[0032]FIG. 23 depicts an example layout similar to the example layoutshown in FIG. 22 with a different keyboard layout.

[0033]FIG. 24 depicts an example layout in which the keyboard isrendered in a window on top of the word recognition area, status/helparea, and tab bar.

[0034]FIG. 25 depicts an example layout with a keypad to the righthaving soft keys such as the delete, end, home, and tab keys.

[0035]FIG. 26 shows multiple additional layouts, including a tools areawith backspace and space soft keys.

[0036]FIG. 27 depicts a TIP in conjunction with a word processor showingtext in the word processing window that was input via the TIP.

DETAILED DESCRIPTION OF THE INVENTION

[0037] The present invention may be more readily described withreference to FIGS. 1-27. FIG. 1 illustrates a schematic diagram of aconventional general-purpose digital computing environment that can beused to implement various aspects of the present invention. In FIG. 1, acomputer 1 DO includes a processing unit 110, a system memory 120, and asystem bus 130 that couples various system components including thesystem memory to the processing unit 110. The system bus 130 may be anyof several types of bus structures including a memory bus or memorycontroller, a peripheral bus, and a local bus using any of a variety ofbus architectures. The system memory 120 includes read only memory (ROM)140 and random access memory (RAM) 150.

[0038] A basic input/output system 160 (BIOS), containing the basicroutines that help to transfer information between elements within thecomputer 100, such as during start-up, is stored in the ROM 140. Thecomputer 100 also includes a hard disk drive 170 for reading from andwriting to a hard disk (not shown), a magnetic disk drive 180 forreading from or writing to a removable magnetic disk 190, and an opticaldisk drive 191 for reading from or writing to a removable optical disk192 such as a CD ROM or other optical media. The hard disk drive 170,magnetic disk drive 180, and optical disk drive 191 are connected to thesystem bus 130 by a hard disk drive interface 192, a magnetic disk driveinterface 193, and an optical disk drive interface 194, respectively.The drives and their associated computer-readable media providenonvolatile storage of computer readable instructions, data structures,program modules and other data for the personal computer 100. It will beappreciated by those skilled in the art that other types of computerreadable media that can store data that is accessible by a computer,such as magnetic cassettes, flash memory cards, digital video disks,Bernoulli cartridges, random access memories (RAMs), read only memories(ROMs), and the like, may also be used in the example operatingenvironment.

[0039] A number of program modules can be stored on the hard disk drive170, magnetic disk 190, optical disk 192, ROM 140 or RAM 150, includingan operating system 195, one or more application programs 196, otherprogram modules 197, and program data 198. A user can enter commands andinformation into the computer 100 through input devices such as akeyboard 101 and pointing device 102. Other input devices (not shown)may include a joystick, game pad, satellite dish, scanner or the like.These and other input devices are often connected to the processing unit110 through a serial port interface 106 that is coupled to the systembus, but may be connected by other interfaces, such as a parallel port,game port or a universal serial bus (USB). Further still, these devicesmay be coupled directly to the system bus 130 via an appropriateinterface (not shown). A monitor 107 or other type of display device isalso connected to the system bus 130 via an interface, such as a videoadapter 108. In addition to the monitor, personal computers typicallyinclude other peripheral output devices (not shown), such as speakersand printers. In a preferred embodiment, a pen digitizer 165 andaccompanying pen or stylus 166 are provided in order to digitallycapture freehand input. Although a direct connection between the pendigitizer 165 and the processing unit 110 is shown, in practice, the pendigitizer 165 may be coupled to the processing unit 110 via a serialport, parallel port or other interface and the system bus 130 as knownin the art. Furthermore, although the digitizer 165 is shown apart fromthe monitor 107, it is preferred that the usable input area of thedigitizer 165 be co-extensive with the display area of the monitor 107.Further still, the digitizer 165 may be integrated in the monitor 107,or may exist as a separate device overlaying or otherwise appended tothe monitor 107. Microphone 167 is coupled to the system bus via a voiceinterface 168 in a well-known manner.

[0040] The computer 100 can operate in a networked environment usinglogical connections to one or more remote computers, such as a remotecomputer 109. The remote computer 109 can be a server, a, router, anetwork PC, a peer device or other common network node, and typicallyincludes many or all of the elements described above relative to thecomputer 100, although only a memory storage device 111 has beenillustrated in FIG. 1. The logical connections depicted in FIG. 1include a local area network (LAN) 112 and a wide area network (WAN)113. Such networking environments are commonplace in offices,enterprise-wide computer networks, intranets and the Internet.

[0041] When used in a LAN networking environment, the computer 100 isconnected to the local networt: 112 through a network interface oradapter 114. When used in a WAN networking environment, the personalcomputer 100 typically includes a modem 115 or other means forestablishing a communications over the wide area network 113, such asthe Internet. The modem 115, which may be internal or external, isconnected to the system bus 130 via the serial port interface 106. In anetworked environment, program modules depicted relative to the personalcomputer 100, or portions thereof, may be stored in the remote memorystorage device.

[0042] It will be appreciated that the network connections shown areexemplary and other techniques for establishing a communications linkbetween the computers can be used. The existence of any of variouswell-known protocols such as TCP/IP, Ethernet, FTP, HTTP and the like ispresumed, and the system can be operated in a client-serverconfiguration to permit a user to retrieve web pages from a web-basedserver. Any of various conventional web browsers can be used to displayand manipulate data on web pages.

[0043]FIG. 2 illustrates a Tablet PC, also referred to herein as atablet and stylus computer, which can be used in accordance with variousaspects of the present invention. Any or all of the features,subsystems, and functions in the system of FIG. 1 can be included in thecomputer of FIG. 2. The computer 201 includes a large display surface202, e.g., a digitizing flat panel display, preferably, a liquid crystaldisplay (LCD) screen, on which a plurality of graphical user interfacewindows 203 is displayed. The display is preferably oriented in“portrait” mode (i.e., vertically), as opposed to “landscape” mode(i.e., horizontally). Using stylus 204, a user can select, highlight,and write on the digitizing display area. Examples of suitabledigitizing display panels include electromagnetic pen digitizers, suchas the Mutoh or Wacom pen digitizers. Other types of pen digitizers,e.g., optical digitizers, may also be used. The computer 201 interpretsmarks made using stylus 204 in order to manipulate data, enter text, andexecute conventional computer application tasks such as spreadsheets,word processing programs, and the like.

[0044] A stylus could be equipped with buttons or other features toaugment its selection capabilities. In one embodiment, a stylus could beimplemented as a “pencil” or “pen”, in which one end constitutes awriting portion and the other end constitutes an “eraser” end, andwhich, when moved across the display, indicates portions of the displayare to be erased. Additionally, a user's own finger could be used forselecting or indicating portions of the displayed image on atouch-sensitive or proximity-sensitive display.

[0045] In accordance with various inventive principles, a Tablet PC userinterface accepts a spectrum of input ranging from entering charactersvia a simple virtual keyboard through single stroke or single characterrecognition. Single stroke refers to using one stroke to symbolize acharacter. The spectrum continues from single character through cursivehandwriting recognition to speech recognition. A user interface inaccordance with the principles of this invention, preferably integratesthese concepts together into a single user interface that allows a userto input text information via various types of input in a seamlesslyintegrated and synchronized manner such that the order in which text isinput by a user in different forms, such as handwritten text and spokentext, is recognized and preserved.

[0046] These text input techniques typically work differently, that isrelatively more or less effectively, for different users. For example,handwriting recognition and speech recognition may work well for someusers and not as well for other users. Of course, any particular user ismore likely to use recognition techniques that work relatively moreefficiently than other techniques in any given circumstance. Forinstance, while speech recognition may work well for a user, it might beinappropriate for a social context such as a meeting with other people.For dictating a letter in a relatively private setting, however, speechrecognition may be more appropriate. Ambient noise may also affect theeffectiveness of speech recognition. In general terms, individual userswill tend to use different input methods at different times.

[0047] Depending upon what the input is, different types of recognitionmay be relatively more or less effective. For instance, cursivehandwriting recognition may be very effective for a textual documentthat includes relatively few special symbols. A symbol recognizer, onthe other hand, will typically be relatively more effective atrecognizing URL's, mathematical equations, and the like.

[0048] Preferably, a user is able to configure various layout settingsof a text input panel (“TIP”) in accordance with various inventiveprinciples. For example, the character recognizer may be shown or hiddenbased upon how the user configures the user interface. Several possiblelayout variations are set forth in the figures and are discussed below.As will be apparent, other suitable layouts could also be used.

[0049]FIG. 3 is a simplified schematic block diagram depicting data flowamong various input recognizers and other components in accordance withcertain inventive principles. Referring to FIG. 3, a soft keyboard 300processes input from a Tablet PC's virtual keyboard/key bar/symbols. Inother words, the soft keyboard 300 processes input, other thanhandwriting, in the form of button clicks and menu choices. A wordrecognizer 302 and a character recognizer 304, on the other hand, aregrouped together within a box drawn in dashed lines to indicate thatboth of these recognizers may be presented together to the user, subjectto the manner in which the user configures the text input panel. Thedashed-line box around soft keyboard 300 indicates that, from the user'sperspective, the soft keyboard is a different part of the user interfacethan the word and character recognizers. A speech recognizer 306translates the user's speech, which may be entered by a user throughmicrophone 167 (FIG. 1), into text.

[0050] A user's goal in using a TIP is typically to input text to anapplication program 308. The soft keyboard is a reliable way to entertext into an application. The user brings up the soft keyboard andactuates it by tapping on keys with the user's pen or stylus. On-screenpresses are translated into actual keystroke commands that an operatingsystem understands.

[0051] The soft keyboard may be sized automatically to approximately thesame size that the word recognizer and character recognizer previouslyoccupied on the display. Alternatively, separate size,, could be storedfor use with the soft keyboard, on the one hand, and the character andword recognizers, on the other hand.

[0052] The soft keyboard, character, word, and speech recognizers eachpass data to a text recognizer/synchronizer 310, which synchronizes andintegrates input from these sources so that if a user inputs somehandwriting and then taps on a soft keyboard key, textrecognizer/synchronizer 310 preserves the order in which the userentered the input from various sources. Text synchronizer 310 preferablyexecutes as a background process. Notwithstanding that the wordrecognizer operates fairly quickly as perceived by the user, thesynchronization mentioned above is desirable because the word recognizertypically takes longer to recognize words than the soft keyboard takesto recognize key presses. Accordingly, if a user writes several words inthe word recognizer area, also referred to as the word pad, then tapsthe space key, the words are preferably recognized first. Then, thespace is inserted into the text following the recognized words, ratherthan ordering the recognized text in the order in which recognition ofthe text was completed, which would not necessarily preserve the orderin which the user entered the text. Speech recognition, like handwritingrecognition, is typically relatively slower than recognition of a softkeyboard press.

[0053] In addition, a user may turn off automatic word recognition,which otherwise preferably occurs after a user-configurable timeoutexpires following a period of inking inactivity. When automatic wordrecognition is turned off, word recognition will not occur until theuser performs a text input panel operation other than writing words.

[0054] The handwriting recognizer 314 is shown within the samedashed-line box in FIG. 3 as the text recognizer/synchronizer 310 toindicate that handwriting recognition and synchronization of textualinput from various sources may both be implemented as part of a process,or execution thread, that preferably executes in the background to allowthe user interface to remain responsive to incoming requests from theuser. Recognition results are then passed from the background textrecognizer/synchronizer 310 to the operating system message router 312,which passes the recognition results, typically in the form of text, tothe application 308.

[0055] Recognition results may optionally be passed to a window forproviding a preview of the recognition results. This preview window isalso referred to as a stage and is depicted in FIG. 3 with referencenumber 316. The stage may pass the recognition results to an operatingsystem message router 312.

[0056] Because the text input panel typically does not have directaccess to various applications running on the user's computer, therecognized text is displayed and a menu of alternate choices that havecome back from the recognizer may also be provided so that the user cancorrect recognition inaccuracies before the text input panel sends thetext to one or more applications. A re-do/re-send button (not shown) maybe provided so that a user can send the same text to multipleapplications or can re-send text that did not reach the intendedapplication.

[0057] The character recognizer preferably recognizes various strokes orgestures as abbreviations for certain characters or special symbols.Some gestures may be carried over from the character recognizer to theword recognizer. For instance, there may be a left-to-right horizontalstroke for the character recognizer that is recognized as a blank spacecharacter and a right-to-left horizontal stroke that is recognized as abackspace. Because spaces and backspaces are entered frequently, thetext input panel preferably recognizes these gestures entered into anempty word recognizer area. Having a vocabulary of gestures that a usercan mix with the user's natural handwriting provides convenience andincreased efficiency to the user.

[0058] A user can preferably have a text input panel displayed and alsoget to see through the text input panel to what's underneath the textinput panel. Because displaying the text input panel in a translucentfashion may be undesirable to some users under certain circumstances,users are preferably able to configure the text input panel to bedisplayed in translucent form never, always, or when the user's penhovers over the text input panel.

[0059] Like the symbol and character recognizer pads shown in variousfigures and discussed above, a gesture pad could be provided. A user'sinking or gestures in such a gesture pad area could be interpreted as agesture as opposed to a symbol or other character. Such a gesture padwould advantageously increase the accuracy of the left-to-right gesturebeing interpreted as a space instead of being erroneously recognized asa hyphen or an underscore, for instance.

[0060] FIGS. 4-27 show various example layouts for a text input panel inaccordance with various inventive principles. Display of these examplelayouts may be performed in a well-known manner for displaying variouscomponents of graphical user interfaces. FIG. 4 depicts several examplelayouts for a slide-out text input panel 401 having three tabs forswitching between input recognizers, namely, Words tab 400, Keyboard tab402, and Characters tab 404. The example layouts shown in FIG. 4 arenumbered in the lower right hand corner of each example layout from 1through 7, with layouts 1 and 2 on the first horizontal row, layouts 3and 4 on the second row, etc. The Words tab 400 corresponds to a wordrecognizer. A symbol button pad 406 is shown in Example layout 1. Thesymbol button pad facilitates entry of symbol characters, such as thesymbols shown on the symbol pad 406 of example layout 1 of FIG. 4.

[0061] A view menu symbolized by the word “View” 408 on the tab barallows a user to select between using a tab, such as the Words tab, witha symbol pad, character recognizer, or none at all, as depicted inexample layouts 1, 2, and 3 of FIG. 4, respectively. The Character tabmay also interact with the view menu in a similar manner. Users cantoggle on/off or swap the symbol pad with the word recognizer 410, asdepicted in example layouts 5 and 7 of FIG. 4, respectively. TheKeyboard 411 may be a simplified “Qwerty” layout without the functionkeys, as shown in example layouts 4 and 6 of FIG. 4. In such a layout,the Fn button 412, which is located in the lower right-hand corner oflayouts 4 and 6, may be used to open a pop-menu of function keys, F1-12.The Symbols menu item 414 in the tab bar may be used to bring up an MRU(most recently used) menu of symbols and/or other prioritized symbols.

[0062] User assistance can be found in a status help area 416 on examplelayouts 1, 2, 3, 5, and 7 of FIG. 4. These example layouts use the areajust below the recognition pads (the area where “word recognizer”appears in layout 1) to give the user, better cues on what each tab doesand how it works. As shown in FIG. 4, layouts 1-3 and 7 include wordrecognizer areas, such as word recognizer area 410, for recognizinghandwritten words, while example layouts 5 and 7 include characterrecognizer areas for recognizing handwritten characters and/or gestures,as described in more detail below.

[0063] The word recognition grid may use a watermark (not shown), whichcould disappear when the pen is hovered over it, for showing users whererecognized handwritten text, also referred to as ink, should be written.The Stage 418, which is an area for displaying a preview of recognizedcharacters, symbols, or words, also has descriptive text 420 thatappears when it is first opened, as shown near the top of example layout3 of FIG. 4.

[0064] The “thumb tab” 422 appearing near the middle from top to bottomon the right side of each example layout in FIG. 4 can be used to “slideout” a text input panel 401 when a user wants to use the text inputpanel. Under such circumstances, while the text input panel is not beingused, only the thumb tab may be displayed to a user. When the userselects, clicks on, or activates the thumb tab, the text input panel canbe displayed, which is referred to as sliding out above. The thumb tabcan be used again after the user is done with the text input panel to“hide” or slide in the text input panel, with the possible exception ofthe thumb tab.

[0065]FIG. 5 depicts several example layouts for the Word tab of FIG. 4,referred to as the Writing tab in FIG. 5. A gray border 500 area may beincluded to focus a user's attention to the edit line areas. Anythingwritten in the gray border area is preferably not recognized or renderedon the screen as ink. The bottom right example layout includes a griddesign 502 for the character recognizer on the right side with discreteareas for capital letters, lower case letters, numbers, and symbolcharacters. The other example layouts of FIG. 5 show various embodimentsof a word recognizer area in accordance with certain inventiveprinciples.

[0066]FIG. 6 depicts a text input panel 401 docked at (i.e., positionedabove and adjacent to) the taskbar 602. Taskbar 602 includes a statusarea 604 at the right side of the taskbar. The text input panel 401 isdepicted as having been launched from the status area 604. The keyboard411 and word recognizer area 410 are presented as tabs in the statusarea 604 of the taskbar 602. This example layout also has a menu bar 606on top with a keybar 608 below. This layout occupies the full width ofthe display screen. The text input panel 401 depicted in FIG. 6 could begrabbed by the top bar and ripped off (i.e., dragged away from) thetaskbar to float like a separate window.

[0067]FIG. 7 depicts the text input panel docked at the taskbar as inFIG. 6 with a “jot” 700 (i.e., character recognizer) selected from themenu bar. As opposed to a word recognition input area, in whichrecognition processing is performed in an attempt to recognize entirewords, the character recognizer 700 is used to input data to arecognition process that attempts to recognize individual characters.

[0068]FIG. 8 depicts the text input panel docked at the taskbar as inFIG. 6 with the keyboard displayed, instead of the word recognizer area410, which is displayed in FIG. 6. As shown in FIG. 8, the keyboard isdisplayed when the keyboard icon in the lower right hand corner isselected.

[0069]FIG. 9 depicts the text input panel docked at the taskbar as inFIG. 6 with a toggle-on button 900 in the lower right-hand corner. Thepop-up menu arrow 902 to the right of the status icon 900 can be used toswitch between the word recognition area 410 and the keyboard 411. TheWriting tab (also referred to above as the Word tab) 400 and theKeyboard tab 402 can also be used to switch between the word recognitionarea 410 and the keyboard 411. The example layout of FIG. 9 includes thekeybar as a button pad 904 to the right of the word recognition area410.

[0070]FIG. 10 depicts the character recognizer area 1000 as a singlepad, as opposed to the separate areas provided in the lower-right-handlayout of FIG. 5 (grid 502), discussed above. The character pad of FIG.10 can be set to recognize capital/upper-case letters, lower-caseletters, or numbers and symbols by selecting the buttons below thecharacter pad 1000.

[0071]FIG. 11 depicts the docked text input panel with a multi-linestage 1100, which as mentioned above, may be used to show a user apreview of the handwriting, keyboard, and/or speech recognition results.

[0072]FIG. 12 depicts a docked text input panel with a full widthkeyboard. In FIG. 12, the Keyboard tab 402 and the Writing tab 400appear to the left-hand side. As will be apparent, the keyboard andwriting tabs could also be placed in other suitable locations. As willalso be apparent other text input panel layouts could also be used. Forexample, FIGS. 13 through 27 depict additional example layouts for atext input panel (“TIP”) in accordance with various inventiveprinciples. FIG. 13 depicts an example layout with a docked TIP, havingthe keybar shown as a keypad on the left-hand side. FIG. 14 depicts anexample layout with a docked TIP having a character recognizer area1000, also referred to as a jot, on the left-hand side. FIG. 15 depictsan example layout with a docked TIP having a multi-line stage 1100 tothe left and Keyboard and Writing tabs 402 and 400 to the right. FIG. 16depicts an example layout with a docked TIP below the taskbar. FIG. 17depicts an example layout with a docked TIP having a double-line wordrecognizer and a double-stacked keybar and stage to the left. FIG. 18depicts an example layout similar to the example layout in FIG. 17 anddepicting a character recognizer area to the left of the word recognizerarea. FIG. 19 depicts an example layout similar to the example layout inFIG. 17 with the keyboard shown instead of the word recognizer. FIG. 20depicts an example layout with a docked TIP having a stage at the bottomand a keybar in the form of a keypad, to the left. FIG. 21 depicts anexample layout similar to the example layout in FIG. 20 with a characterrecognizer area shown. FIG. 22 depicts an example layout with a keyboardshown and a stage area at the bottom. FIG. 23 depicts an example layoutsimilar to the example layout shown in FIG. 22 with a different keyboardlayout including an inverted “T” for up, down, left, and right arrowsand a number pad similar to a number pad commonly present on aconventional physical keyboard. FIG. 24 depicts an example layout inwhich the keyboard is rendered in a window on top of the wordrecognition area, status/help area, and tab bar. FIG. 26 shows multipleadditional layouts, including a tools area with backspace and space softkeys. FIG. 25 depicts an example layout with a keypad to the righthaving soft keys such as the delete, tab, home, end, arrow, and enterkeys. As will be apparent, other suitable keys and layouts could also beused.

[0073] As mentioned above, a user's goal in using a text input panel, inaccordance with various inventive principles, would often be enteringtextual input into an application executing on the user's computer. FIG.27 depicts a text input panel in conjunction with a word processorshowing text that has been input to the word processing application viathe text input panel.

[0074] What has been described above is merely illustrative of theapplication of the principles of the present invention. Those skilled inthe art can implement other arrangements and methods without departingfrom the spirit and scope of the present invention. Any of the methodsof the invention can be implemented in software that can be stored oncomputer disks or other computer-readable media. No claim should beinterpreted to be in means-plus-function format.

We claim:
 1. A system for accepting disparate types of computer userinput, the system comprising: a user interface including a handwritingrecognition area and a soft keyboard; a handwriting recognizer forrecognizing handwritten text input by a user via the handwritingrecognition area; and a text synchronizer for synchronizing text inputby a user via the handwriting recognition area and via the soft keyboardby recognizing and preserving an order in which a user inputs text viathe handwriting recognition area and the soft keyboard.
 2. The system ofclaim 1 wherein: the user interface further includes a speechrecognizer; and the text synchronizer recognizes and preserves an orderin which the user inputs text via the speech recognizer, via thehandwriting recognition area, and via the soft keyboard.
 3. The systemof claim 1 wherein the handwriting recognition area includes a wordrecognition area.
 4. The system of claim 3 wherein the handwritingrecognition area includes a character recognition area.
 5. The system ofclaim 4 wherein the character recognition area operates in a characterrecognition mode selected from the group consisting of: recognition oflowercase letters, recognition of uppercase letters, and recognition ofcharacters other than letters.
 6. The system of claim 4 wherein thecharacter recognition area includes separate areas for recognizinglowercase letters, uppercase letters, numbers, and characters other thanletters and numbers, respectively.
 7. The system of claim 1 wherein theuser interface includes a plurality of tabs for switching from one to atleast one other of the soft keyboard, the word recognizer, and thecharacter recognizer.
 8. The system of claim 1 wherein the userinterface includes a keybar or keypad having at least one key selectedfrom the group consisting of: left arrow, right arrow, backspace,delete, space, end, home, enter, tab, and escape.
 9. The system of claim1 wherein the user interface includes a symbol pad having at least onekey for entering symbols that are characters other than letters andnumbers.
 10. The system of claim 1 wherein the handwriting recognitionarea includes a gesture area For recognizing handwritten gestures orabbreviations.
 11. The system of claim 1 wherein the user interfaceincludes a stage area for displaying a preview of recognized text inputby the user.
 12. A method of accepting disparate types of computer userinput, the method comprising: recognizing handwritten user input toprovide recognized handwritten user input; recognizing soft keyboardpresses to provide recognized soft keyboard presses; integrating textinput by a user by synchronizing the recognized handwritten user inputand the recognized soft keyboard presses.
 13. The method of claim 12further comprising: recognizing spoken user input to provide recognizedspoken user input; and integrating text input by a user by synchronizingthe recognized spoken user input, the recognized handwritten user input,and the recognized soft keyboard presses.
 14. The method of claim 13further comprising: displaying a preview of the recognized text input bya user.
 15. The method of claim 13 further comprising: passing therecognized text to an operating system message router.
 16. The method ofclaim 13 further comprising: passing the recognized text to anapplication program.
 17. The method of claim 13 further comprising:recognizing words inked by a user onto a word recognition area.
 18. Themethod of claim 17 further comprising: recognizing characters inked by auser onto a character recognition area.
 19. The method of claim 18further comprising: recognizing gestures inked by a user onto thecharacter recognition area.
 20. The method of claim 19 furthercomprising: recognizing gestures inked by a user onto the wordrecognition area.
 21. A text input panel for accepting disparate typesof computer user input, the text input panel comprising: a keyboard tabthat, when selected, activates a soft keyboard for accepting softkeyboard presses; a word recognizer tab that, when selected, activates aword recognizer area for accepting user input in the form of handwrittenwords; and a character recognizer tab that, when selected, activates acharacter recognizer area for accepting user input in the form ofhandwritten characters.
 22. The text input panel of claim 21 wherein thetext input panel includes a tab for alternately hiding and displayingthe text input panel.
 23. The text input panel of claim 21 wherein thetext input panel is displayed translucently such that a user can seethrough the text input panel.